Text Summarization

Part 2, Chapter 15


In this chapter, we'll develop the text summarization piece of our application.


Newspaper3k

Our objective is to create a real-time text summarization service used for creating article summaries from a given URL.

We'll need to use a web scraping library to extract the text of an article and then use a machine learning library to generate the actual summary. We could roll our own solution with say Beautiful Soup (for scraping) and Bart (for summarizing), but there's a library that will do both: Newspaper3k.

So, start by adding the library to the requirements file:

newspaper3k==0.2.8

Install:

$ docker-compose up -d --build

Summarizer

Add a new file called project/app/summarizer.py:

# project/app/summarizer.py


from newspaper import Article


def generate_summary(url: str) -> str:
    article = Article(url)
    article.download()
    article.parse()
    article.nlp()
    return article.summary

That's it.

After creating a new Article instance, we downloaded the given URL's HTML, extracted the meaningful content (via parse()) and the relevant keywords (via nlp()), and then generated a summary.

For more, review the Quickstart guide from the official docs.

The nlp method requires the punkt tokenizer from the Natural Language Toolkit (NLTK), so update summarizer.py like so to install it:

# project/app/summarizer.py


import nltk
from newspaper import Article


def generate_summary(url: str) -> str:
    article = Article(url)
    article.download()
    article.parse()

    try:
        nltk.data.find("tokenizers/punkt")
    except LookupError:
        nltk.download("punkt")
    finally:
        article.nlp()

    return article.summary

Next, update the post function in project/app/api/crud.py to generate the summary when a new URL is added:

async def post(payload: SummaryPayloadSchema) -> int:
    article_summary = generate_summary(payload.url)
    summary = TextSummary(url=payload.url, summary=article_summary)
    await summary.save()
    return summary.id

Don't forget the import:

from app.summarizer import generate_summary

Test it out:

$ http --json POST http://localhost:8004/summaries/ url=http://testdriven.io

HTTP/1.1 201 Created
content-length: 37
content-type: application/json
date: Mon, 27 Dec 2021 21:57:44 GMT
server: uvicorn

{
    "id": 1,
    "url": "http://testdriven.io"
}

$ http GET http://localhost:8004/summaries/1/

HTTP/1.1 200 OK
content-length: 779
content-type: application/json
date: Mon, 27 Dec 2021 21:58:23 GMT
server: uvicorn

{
    "created_at": "2021-07-08T00:10:39.390969+00:00",
    "id": 1,
    "summary": "Our courses and tutorials teach practical application of popular tech used by both enterprise and startups.\nYou will quickly become effective in your chosen stack, as we provide pragmatic examples and detailed walkthroughs of the workings of each technology.\nThrough our courses, you will not only become more comfortable with specific tools and technologies like AWS ECS, Docker, and Flask, to name a few -- but you will also gain skills necessary to contribute to a team or launch your own exciting project.\nEvery tool taught in our courses is supported by large communities and is in high-demand by hiring managers around the globe.\nWe teach tools we love and use every day.",
    "url": "http://testdriven.io"
}

Background Task

The generate_summary function introduces a blocking operation into the post function. Since both article.parse() and article.nlp() are expensive operations, let's move the entire generate_summary function out of the request/response flow by running it as a background task.

First, update post:

async def post(payload: SummaryPayloadSchema) -> int:
    summary = TextSummary(url=payload.url, summary="")
    await summary.save()
    return summary.id

Next, update create_summary:

@router.post("/", response_model=SummaryResponseSchema, status_code=201)
async def create_summary(payload: SummaryPayloadSchema, background_tasks: BackgroundTasks) -> SummaryResponseSchema:
    summary_id = await crud.post(payload)

    background_tasks.add_task(generate_summary, summary_id, payload.url)

    response_object = {"id": summary_id, "url": payload.url}
    return response_object

Make sure to import both BackgroundTasks and generate_summary:

from fastapi import APIRouter, HTTPException, Path, BackgroundTasks

from app.summarizer import generate_summary

Finally, update generate_summary like so:

# project/app/summarizer.py


import nltk
from newspaper import Article

from app.models.tortoise import TextSummary


async def generate_summary(summary_id: int, url: str) -> None:
    article = Article(url)
    article.download()
    article.parse()

    try:
        nltk.data.find("tokenizers/punkt")
    except LookupError:
        nltk.download("punkt")
    finally:
        article.nlp()

    summary = article.summary

    await TextSummary.filter(id=summary_id).update(summary=summary)

Now, the summary is generated after the response is sent back to the client. Once generated, the database is updated. To test, add asyncio.sleep to generate_summary:

# project/app/summarizer.py


import asyncio

import nltk
from newspaper import Article

from app.models.tortoise import TextSummary


async def generate_summary(summary_id: int, url: str) -> None:
    article = Article(url)
    article.download()
    article.parse()

    try:
        nltk.data.find("tokenizers/punkt")
    except LookupError:
        nltk.download("punkt")
    finally:
        article.nlp()

    summary = article.summary

    await asyncio.sleep(10)

    await TextSummary.filter(id=summary_id).update(summary=summary)

Then, add a new summary:

$ http --json POST http://localhost:8004/summaries/ url=http://testdriven.io

HTTP/1.1 201 Created
content-length: 34
content-type: application/json
date: Tue, 28 Dec 2021 00:09:22 GMT
server: uvicorn

{
    "id": 5,
    "url": "http://testdriven.io"
}

Get the summary:

$ http GET http://localhost:8004/summaries/5/

HTTP/1.1 200 OK
content-length: 134
content-type: application/json
date: Tue, 28 Dec 2021 00:09:39 GMT
server: uvicorn

{
    "created_at": "2020-05-10T15:59:55.098074",
    "id": 5,
    "summary": "",
    "url": "http://testdriven.io"
}

Wait about ten seconds before getting the summary again:

$ http GET http://localhost:8004/summaries/5/

HTTP/1.1 200 OK
content-length: 134
content-type: application/json
date: Tue, 28 Dec 2021 00:09:52 GMT
server: uvicorn

{
    "created_at": "2020-05-10T15:59:55.098074",
    "id": 5,
    "summary": "Our courses and tutorials teach practical application of popular tech used by both enterprise and startups.\nYou will quickly become effective in your chosen stack, as we provide pragmatic examples and detailed walkthroughs of the workings of each technology.\nThrough our courses, you will not only become more comfortable with specific tools and technologies like AWS ECS, Docker, and Flask, to name a few -- but you will also gain skills necessary to contribute to a team or launch your own exciting project.\nEvery tool taught in our courses is supported by large communities and is in high-demand by hiring managers around the globe.\nWe teach tools we love and use every day.",
    "url": "http://testdriven.io"
}

Be sure to remove asyncio.sleep once done.

Tests

Do the tests pass?

$ docker-compose exec web python -m pytest

You should see six failures:

>           raise ArticleException('Article `download()` failed with %s on URL %s' %
                  (self.download_exception_msg, self.url))
E           newspaper.article.ArticleException: Article `download()` failed with HTTPSConnectionPool(host='foo.bar', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f00e5582e50>: Failed to establish a new connection: [Errno -2] Name or service not known')) on URL https://foo.bar

Let's use monkeypatch again.

Example:

def test_create_summary(test_app_with_db, monkeypatch):
    def mock_generate_summary(summary_id, url):
        return None
    monkeypatch.setattr(summaries, "generate_summary", mock_generate_summary)

    response = test_app_with_db.post(
        "/summaries/", data=json.dumps({"url": "https://foo.bar"})
    )

    assert response.status_code == 201
    assert response.json()["url"] == "https://foo.bar"

Add the import:

from app.api import summaries

Fix the other failing tests on your own.

Run Flake8, Black, and isort:

$ docker-compose exec web flake8 .
$ docker-compose exec web black .
$ docker-compose exec web isort .

Commit your code and push it to GitHub. Test the summarizer on Heroku.




✓ Mark as Completed